<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Epipolar Geometry</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d6/d00/tutorial_py_root.html">OpenCV-Python Tutorials</a></li><li class="navelem"><a class="el" href="../../d9/db7/tutorial_py_table_of_contents_calib3d.html">Camera Calibration and 3D Reconstruction</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Epipolar Geometry </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h2>Goal </h2>
<p>In this section,</p>
<ul>
<li>We will learn about the basics of multiview geometry</li>
<li>We will see what is epipole, epipolar lines, epipolar constraint etc.</li>
</ul>
<h2>Basic Concepts </h2>
<p>When we take an image using pin-hole camera, we loose an important information, ie depth of the image. Or how far is each point in the image from the camera because it is a 3D-to-2D conversion. So it is an important question whether we can find the depth information using these cameras. And the answer is to use more than one camera. Our eyes works in similar way where we use two cameras (two eyes) which is called stereo vision. So let's see what OpenCV provides in this field.</p>
<p>(<em>Learning OpenCV</em> by Gary Bradsky has a lot of information in this field.)</p>
<p>Before going to depth images, let's first understand some basic concepts in multiview geometry. In this section we will deal with epipolar geometry. See the image below which shows a basic setup with two cameras taking the image of same scene.</p>
<div class="image">
<img src="../../epipolar.jpg" alt="epipolar.jpg"/>
<div class="caption">
image</div></div>
<p> If we are using only the left camera, we can't find the 3D point corresponding to the point \(x\) in image because every point on the line \(OX\) projects to the same point on the image plane. But consider the right image also. Now different points on the line \(OX\) projects to different points ( \(x&#39;\)) in right plane. So with these two images, we can triangulate the correct 3D point. This is the whole idea.</p>
<p>The projection of the different points on \(OX\) form a line on right plane (line \(l&#39;\)). We call it <b>epiline</b> corresponding to the point \(x\). It means, to find the point \(x\) on the right image, search along this epiline. It should be somewhere on this line (Think of it this way, to find the matching point in other image, you need not search the whole image, just search along the epiline. So it provides better performance and accuracy). This is called <b>Epipolar Constraint</b>. Similarly all points will have its corresponding epilines in the other image. The plane \(XOO&#39;\) is called <b>Epipolar Plane</b>.</p>
<p>\(O\) and \(O&#39;\) are the camera centers. From the setup given above, you can see that projection of right camera \(O&#39;\) is seen on the left image at the point, \(e\). It is called the <b>epipole</b>. Epipole is the point of intersection of line through camera centers and the image planes. Similarly \(e&#39;\) is the epipole of the left camera. In some cases, you won't be able to locate the epipole in the image, they may be outside the image (which means, one camera doesn't see the other).</p>
<p>All the epilines pass through its epipole. So to find the location of epipole, we can find many epilines and find their intersection point.</p>
<p>So in this session, we focus on finding epipolar lines and epipoles. But to find them, we need two more ingredients, <b>Fundamental Matrix (F)</b> and <b>Essential Matrix (E)</b>. Essential Matrix contains the information about translation and rotation, which describe the location of the second camera relative to the first in global coordinates. See the image below (Image courtesy: Learning OpenCV by Gary Bradsky):</p>
<div class="image">
<img src="../../essential_matrix.jpg" alt="essential_matrix.jpg"/>
<div class="caption">
image</div></div>
<p> But we prefer measurements to be done in pixel coordinates, right? Fundamental Matrix contains the same information as Essential Matrix in addition to the information about the intrinsics of both cameras so that we can relate the two cameras in pixel coordinates. (If we are using rectified images and normalize the point by dividing by the focal lengths, \(F=E\)). In simple words, Fundamental Matrix F, maps a point in one image to a line (epiline) in the other image. This is calculated from matching points from both the images. A minimum of 8 such points are required to find the fundamental matrix (while using 8-point algorithm). More points are preferred and use RANSAC to get a more robust result.</p>
<h2>Code </h2>
<p>So first we need to find as many possible matches between two images to find the fundamental matrix. For this, we use SIFT descriptors with FLANN based matcher and ratio test. </p><div class="fragment"><div class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</div><div class="line"><span class="keyword">import</span> cv2 <span class="keyword">as</span> cv</div><div class="line"><span class="keyword">from</span> matplotlib <span class="keyword">import</span> pyplot <span class="keyword">as</span> plt</div><div class="line"></div><div class="line">img1 = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">cv.imread</a>(<span class="stringliteral">&#39;myleft.jpg&#39;</span>,0)  <span class="comment">#queryimage # left image</span></div><div class="line">img2 = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">cv.imread</a>(<span class="stringliteral">&#39;myright.jpg&#39;</span>,0) <span class="comment">#trainimage # right image</span></div><div class="line"></div><div class="line">sift = cv.SIFT_create()</div><div class="line"></div><div class="line"><span class="comment"># find the keypoints and descriptors with SIFT</span></div><div class="line">kp1, des1 = sift.detectAndCompute(img1,<span class="keywordtype">None</span>)</div><div class="line">kp2, des2 = sift.detectAndCompute(img2,<span class="keywordtype">None</span>)</div><div class="line"></div><div class="line"><span class="comment"># FLANN parameters</span></div><div class="line">FLANN_INDEX_KDTREE = 1</div><div class="line">index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)</div><div class="line">search_params = dict(checks=50)</div><div class="line"></div><div class="line">flann = <a class="code" href="../../dc/de2/classcv_1_1FlannBasedMatcher.html">cv.FlannBasedMatcher</a>(index_params,search_params)</div><div class="line">matches = flann.knnMatch(des1,des2,k=2)</div><div class="line"></div><div class="line">pts1 = []</div><div class="line">pts2 = []</div><div class="line"></div><div class="line"><span class="comment"># ratio test as per Lowe&#39;s paper</span></div><div class="line"><span class="keywordflow">for</span> i,(m,n) <span class="keywordflow">in</span> enumerate(matches):</div><div class="line">    <span class="keywordflow">if</span> m.distance &lt; 0.8*n.distance:</div><div class="line">        pts2.append(kp2[m.trainIdx].pt)</div><div class="line">        pts1.append(kp1[m.queryIdx].pt)</div></div><!-- fragment --><p> Now we have the list of best matches from both the images. Let's find the Fundamental Matrix. </p><div class="fragment"><div class="line">pts1 = np.int32(pts1)</div><div class="line">pts2 = np.int32(pts2)</div><div class="line">F, mask = <a class="code" href="../../d9/d0c/group__calib3d.html#gae850fad056e407befb9e2db04dd9e509">cv.findFundamentalMat</a>(pts1,pts2,cv.FM_LMEDS)</div><div class="line"></div><div class="line"><span class="comment"># We select only inlier points</span></div><div class="line">pts1 = pts1[mask.ravel()==1]</div><div class="line">pts2 = pts2[mask.ravel()==1]</div></div><!-- fragment --><p> Next we find the epilines. Epilines corresponding to the points in first image is drawn on second image. So mentioning of correct images are important here. We get an array of lines. So we define a new function to draw these lines on the images. </p><div class="fragment"><div class="line"><span class="keyword">def </span>drawlines(img1,img2,lines,pts1,pts2):</div><div class="line">    <span class="stringliteral">&#39;&#39;&#39; img1 - image on which we draw the epilines for the points in img2</span></div><div class="line"><span class="stringliteral">        lines - corresponding epilines &#39;&#39;&#39;</span></div><div class="line">    r,c = img1.shape</div><div class="line">    img1 = <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#ga397ae87e1288a81d2363b61574eb8cab">cv.cvtColor</a>(img1,cv.COLOR_GRAY2BGR)</div><div class="line">    img2 = <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#ga397ae87e1288a81d2363b61574eb8cab">cv.cvtColor</a>(img2,cv.COLOR_GRAY2BGR)</div><div class="line">    <span class="keywordflow">for</span> r,pt1,pt2 <span class="keywordflow">in</span> zip(lines,pts1,pts2):</div><div class="line">        color = tuple(np.random.randint(0,255,3).tolist())</div><div class="line">        x0,y0 = map(int, [0, -r[2]/r[1] ])</div><div class="line">        x1,y1 = map(int, [c, -(r[2]+r[0]*c)/r[1] ])</div><div class="line">        img1 = <a class="code" href="../../d6/d6e/group__imgproc__draw.html#ga7078a9fae8c7e7d13d24dac2520ae4a2">cv.line</a>(img1, (x0,y0), (x1,y1), color,1)</div><div class="line">        img1 = <a class="code" href="../../d6/d6e/group__imgproc__draw.html#gaf10604b069374903dbd0f0488cb43670">cv.circle</a>(img1,tuple(pt1),5,color,-1)</div><div class="line">        img2 = <a class="code" href="../../d6/d6e/group__imgproc__draw.html#gaf10604b069374903dbd0f0488cb43670">cv.circle</a>(img2,tuple(pt2),5,color,-1)</div><div class="line">    <span class="keywordflow">return</span> img1,img2</div></div><!-- fragment --><p> Now we find the epilines in both the images and draw them. </p><div class="fragment"><div class="line"><span class="comment"># Find epilines corresponding to points in right image (second image) and</span></div><div class="line"><span class="comment"># drawing its lines on left image</span></div><div class="line">lines1 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga19e3401c94c44b47c229be6e51d158b7">cv.computeCorrespondEpilines</a>(pts2.reshape(-1,1,2), 2,F)</div><div class="line">lines1 = lines1.reshape(-1,3)</div><div class="line">img5,img6 = drawlines(img1,img2,lines1,pts1,pts2)</div><div class="line"></div><div class="line"><span class="comment"># Find epilines corresponding to points in left image (first image) and</span></div><div class="line"><span class="comment"># drawing its lines on right image</span></div><div class="line">lines2 = <a class="code" href="../../d9/d0c/group__calib3d.html#ga19e3401c94c44b47c229be6e51d158b7">cv.computeCorrespondEpilines</a>(pts1.reshape(-1,1,2), 1,F)</div><div class="line">lines2 = lines2.reshape(-1,3)</div><div class="line">img3,img4 = drawlines(img2,img1,lines2,pts2,pts1)</div><div class="line"></div><div class="line">plt.subplot(121),plt.imshow(img5)</div><div class="line">plt.subplot(122),plt.imshow(img3)</div><div class="line">plt.show()</div></div><!-- fragment --><p> Below is the result we get:</p>
<div class="image">
<img src="../../epiresult.jpg" alt="epiresult.jpg"/>
<div class="caption">
image</div></div>
<p> You can see in the left image that all epilines are converging at a point outside the image at right side. That meeting point is the epipole.</p>
<p>For better results, images with good resolution and many non-planar points should be used.</p>
<h2>Additional Resources </h2>
<h2>Exercises </h2>
<ol type="1">
<li>One important topic is the forward movement of camera. Then epipoles will be seen at the same locations in both with epilines emerging from a fixed point. <a href="http://answers.opencv.org/question/17912/location-of-epipole/">See this discussion</a>.</li>
<li>Fundamental Matrix estimation is sensitive to quality of matches, outliers etc. It becomes worse when all selected matches lie on the same plane. <a href="http://answers.opencv.org/question/18125/epilines-not-correct/">Check this discussion</a>. </li>
</ol>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:37 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
